As the accumulated knowledge of humankind continues to grow, and the ease of access to information rapidly increases as it has with the advent of the personal computer (PC), the world-wide web and search engines such as the Google® search engine operated by Google, Inc. of Mountain View, Calif., it is common for people to encounter large amounts of information and data that they may wish to store and access later. This information may be in the form of data (such as numerical information or a document containing such information; a calendar and/or its events; a to do list and/or its tasks; or a contact list and/or its constituent contacts), textual form, graphical form (such as photographs, illustrations, or diagrams), audio form (such as music, audio books, podcasts or recorded teleconferences or lectures), video form, or some multi-media combination of these forms (such as a presentation or a mixed-media web page), or other type of document or group of documents.
As the need has grown for users to quickly and easily store, organize, and retrieve information, various methods and tools have become available to facilitate different types of information storage and retrieval. When web browsers such as the Netscape web browser operated by Netscape Communications Corporation of Mountain View, Calif. and the Internet Explorer web browser operated by Microsoft Corporation of Redmond Wash. first became available, they had a feature that enabled users to bookmark web pages which were of interest, and to which the user might want to return later. Bookmarks could be organized into folders to make it easier to find bookmarks later, but many users have found the process of organizing bookmarks into folders too time-consuming for all but their most frequently used bookmarks and/or generated a huge general folder full of bookmarks.
Once a user has built up hundreds of bookmarks, it can be challenging to locate a relevant bookmark after it has been saved. Even early versions of browsers generally allowed people to search bookmarks for a word or phrase, and this search capability provides an alternate way to locate a bookmark in a large collection of bookmarks. Unfortunately, to benefit from this method of locating a bookmark, the title of the bookmark needs to contain the word or phrase that is to be searched. Many people don't want to edit the title of a bookmark to include all the keywords they might use for a search to find the bookmark later, so modern browsers also include a separate keyword field for each bookmark in addition to the title field. These keywords have come to be known as tags. This browser feature allows users to type such tags in the form of words and phrases (separated by commas) into the tag field at the time of bookmarking a page, and search on these tags later when wanting to find a bookmark. This additional capability of tagging bookmarks has brought a slight advance in the searchability of bookmarks, but many people still find this interface (and the tagging process itself) cumbersome and time consuming. Thus, users tend not to take the time to type in tags when bookmarking. There is a need for a more natural and easier-to-use way to tag bookmarks in browsers.
In recent years, various note taking software has become available which facilitates storing many types of media in one big bucket (such as a single file or database), where each piece of information stored in the bucket may be text or graphical. The Evernote® database sold by Parascript, LLC of Boulder, Colo. lets users capture notes, images, audio files, links, and web pages in a completely freeform database. Users can tag each entry, organize items into notebooks, and search for tags or text. Users can even search for text that appears in images. Like the bookmarking feature of modern browsers, the Evernote® database offers users the option of typing tags into a dedicated field associated with each item of information to be stored, to ease the process of searching for the stored item later. There are several problems however, with users typing in their own tags. One problem is that this process places a high cognitive load on the user: It requires a great deal of mental energy and time from the user to do properly. For example, the process demands that the user try to quickly bring to mind all the ways a piece of information could or should be tagged, including various ways the user (or others) may eventually seek the information when searching for it later. And, while some desired tags may come to mind easily, others may not. Furthermore, it takes time to think up a list of tags that might prove useful later, and depending on the user's state of mind, the list of tags that come to mind may vary from quite comprehensive to substantially less comprehensive. Another problem is errors: one could easily mistype a tag while saving information, and then be unable to find the stored information by searching on the properly typed tag. As a result of these issues, few people end up using the tagging capability of modern browsers. There is a need for innovative methods that provide rapid tagging capabilities that are less error-prone, and aid users in bringing to mind a comprehensive set of appropriate tags.
In recent years, note-taking software that integrates with web browsers has become available. This software allows users to highlight portions of web pages (including both text and graphics) that are of interest, and save the material of interest to the user's PC hard drive or to a server on the web. One such piece of software had been sold under the trademark iLighter™; when a user highlights and saves material from the web using the iLighter™ software, text notes (which may include tags) may be saved along with the information of interest. When saved information is subsequently recalled, the iLighter™ software allowed the user to expand the displayed material to include the entire web page from which the material of interest was originally taken. The tagging features of the iLighter™ software suffered from the same limitations as the tagging features of modern browsers, in that the user must think up and type in the tags when saving the material of interest.
Tagging is also a current area of research in the field of Semantic Computing at the University of California at Berkeley, which addresses the derivation and matching of the semantics of computational content to that of naturally expressed user intentions in order to retrieve, manage, manipulate or even create content, where content may be any material including video, audio, text, processes, services, hardware and networks. In a paper entitled “Lightweight Tagging Expands Information and Activity Management Practices” (Proceedings of the ACM CHI 2009 Conference on Human Factors in Computing Systems, April 2009, by Gerard Oleksik, Max L. Wilson, Craig Tashman, Eduarda Mendes Rodrigues, Gabriella Kazai, Gavin Smyth, Natasa Milic-Frayling, and Rachel Jones) experiments are disclosed to have been done with software called TAGtivity™, which allows users to tag files by dragging them and dropping them on the TAGtivity icon, and also allows users to tag web pages within a web browser, through a TAGtivity menu. TAGtivity expands on modern browsers' tagging function, by allowing users to either type in tags, or choose from a menu of existing tags, which can be sorted either alphabetically or by frequency of use.
Although later retrieval of stored information is aided by tagging systems such as the bookmark-tagging feature of modern web browsers such as Firefox 3, tagging software such as TAGtivity, and the tagging features of note-taking software such as Evernote, these tagging systems do not aid users in browsing stored information. If users take the time to store bookmarks in folders or to store notes and other information in a database, the folder structure or database structure allow for browsing, but today's tagged information storage systems do not easily facilitate browsing. There is a need for innovative associative linking systems that not only provide the search and retrieval capabilities of modern tagged information storage systems, but that also provide browsing capabilities.
Associative linking may be defined as a means of associating two or more information elements with one another by means of pointer-style links made between graphical or textual representations of such elements on a computer or other device screen using a pointing device, keyboard and/or other means. Associative linking should not be confused with hypertext linking, as the latter implies direct transportation from one page of information to another, or from one region on a web page to an area on another web page. Nor should associative linking be confused with creating aliases, since an alias is simply a textual or iconic element that points to a file or application. By contrast, associative linking enables relationships to be made between multiple information elements. These multiple, associated information elements or data elements may then be aggregated and represented in proximity to the original data or information element, displaying their association with the original data or information element. Associative linking also implies multiple means of visualization to view the associated data or information elements in different ways, depending on the task and the nature of the user. Associative linking more closely resembles the way humans remember information (associatively, or through association with other ideas as well as association with people, places, events, emotions, and other external and internal stimuli) than do existing indexing or computer storage systems. Thus, associative linking provides a framework for enabling users to recall information from their memory with a higher degree of accuracy and speed.
The associative nature of human memory explains why, for example, a graphical user interface that contains windows, icons, menus and pointers (such as the Apple Mac OS or Microsoft Windows) has been shown to be easier to learn and use than command-line interfaces (such as Microsoft DOS or traditional UNIX). Put simply, in a command-line interface, the user must recall all the commands (directions) to give to the computer by memory. In contrast, a graphical user interface with menus enables users to recognize the commands instead of recall them. This difference of between recognition and recall has been shown to place a much lower cognitive demand on the user, thus reducing the time and rote memorization it takes to learn to use the computer system. But beyond rudimentary support for recognition instead of recall, modern graphical user interfaces offer little support for the fact that human memory is fundamentally associative in nature.
Users may sometimes try to make use of the associative nature of their memory, despite the paucity of support for it in modern computer systems. For example, the modern graphical user interfaces of Microsoft Windows, Apple Macintosh and other operating systems enable users to create and name folders for their files, represented either textually or iconically, name their files, and place their files in the folders. If users are careful to label their files and folders clearly, they may use the structure and naming conventions they set up to assist them in tracking down older files they seek. Too often, however, files and folders are not labeled clearly enough to withstand the fading nature of memory, the aggregate nature of multiple drafts, and the accretive process of more and more files and folders confusing the very users that constructed these data storage structures in the first place. The result is often time spent, unsuccessfully searching these data structures over and over for a file that eludes the user.
In recent years, the producers of modern operating systems and others have grown to appreciate the difficulty users have in tracking down their own files. Apple Computer, Inc. of Cupertino, Calif. released the Spotlight™ search feature for its Mac OS, Microsoft Corporation of Redmond, Wash. followed with enhanced search of its own, and Google, Inc. released Google Desktop, a search engine dedicated to finding files on one's own machine. These desktop search systems are capable of searching for words or phrases within files, and not just for names of files. Such systems provide some relief for users, but not nearly enough, since users of these systems are still limited by the same issues that have hampered users searching for bookmarks on their web browsers: They require the user to remember details of files that they barely remember in the first place. Meanwhile, users may continue to remember many different kinds of details about the files, such as where and when they encountered them, with whom they were discussed, etc. There is a need for innovative technologies that facilitate users finding things they have saved, faster and with greater ease than is facilitated by the state of the art in information retrieval.
In recent years, Apple Computer, Inc.'s free iTunes desktop software application has enabled its users to maintain a library of digital music files that feature some aspects of associative linking. Users of iTunes can create and name one or more playlists that are lists of audio files that the user may wish to play for various different occasions. Users of iTunes can, for example, specify a list of songs to be played while exercising, another list to be played while studying, and yet another list to be played at parties. Playlists may be created to match a mood, or showcase a favorite artist or composer, or provide background music for any imaginable occasion. Songs in an iTunes library can belong to multiple playlists, yet remain browsable in the application's master Library. It should be noted that the act of creating or editing playlists is often considered by users to be a form of entertainment in and of itself. It's an act of self-expression in that it is creative to mix and match music together. Song names within a playlist may also be browsed in iTunes by selecting the name of a playlist, which displays the list of songs contained in that playlist. Song titles in iTunes may be seen in multiple contexts, viewed by artist, by album, by date released, or by a whole host of other criteria.
iTunes playlists provide a limited kind of support for associative linking, by enabling a user to find a song based on many user-specified criteria that may involve the user's emotions, such as its context in belonging to one or more playlists specified by the user. Still, when a user is trying to find a particular song, iTunes fails to provide direct support for associative linking by failing to help users who might not remember other details about the song, but remember the person they were with when they heard it, for example, or the location they were in when they were listening to it, or the time of day it was when they heard it, etc. Further, while Apple and others have worked to incorporate elements of iTunes into other applications (such as being able to browse files in the Macintosh operating system using iTunes' Cover Flow, for example), data silos remain, e.g., audio files remain with other audio files, pictures with other pictures, contacts stay in their contact lists, and so on. Associative linking between these data silos remains absent, the walls between data silos stay up, and users have more difficulty than ever finding what they are looking for.
Scientists and engineers have been unable to fully address these basic issues of information retrieval, even though the need has long been recognized. See As We May Think, by Vannevar Bush, Atlantic Monthly, July 1945 (discussing the challenges inherent in retrieving information from an indexing system while the human mind “operates by association”).
Today's relational databases are an advance over the single-index model in use in Vannevar Bush's time. The relational database structure enables users to perform searches based on a wide variety of fields, and even perform searches using Boolean operators. A modern address book software program, for example, may be searched for a name, address, city, state, zip code, phone number, or email address. Still, there is currently no method to easily browse through pictures associated with addresses or cities, or to browse through a list of wines associated with the friends and relatives with whom the user tasted them.
There remains a need, therefore, for systems and methods for a cohesive scalable technique for users of an information system to easily browse their information.